.. _Using NeurEco with MATLAB:

Tutorial: using NeurEco with MATLAB
=========================================================

The following section will use the test case :std:ref:`Energy consumption test case`. This test case is delivered with the NeurEco installation package.

.. note::
    This was tested on Matlab 2017a and newer versions. A python 3 must be installed with a numpy package installed and the NeurEco package installed.
	
.. note::
    This tutorial is working with a **Regression** problem. The calls should be adapted when working with other solutions according to their Python API:
	
    * :std:ref:`Tabular Regression with the Python API`
	
    * :std:ref:`Tabular Compression with the Python API`

    * :std:ref:`Tabular Classification with the Python API`
	
    * :std:ref:`Discrete Dynamic with the Python API`

The first thing to do is create a new directory and place the data files in it. Then change MATLAB working directory to the one just created.

If multiple python environments are available, specify which version of python 3 will be used by running the following MATLAB command:

.. code-block:: matlab 

    pyversion(full path to the python executable of the environment to use) 

the next step is to load the data:

.. code-block:: matlab

    x_test = dlmread('x_test.csv', ';', 1, 0);
    y_test = dlmread('y_test.csv', ';', 1, 0);
    x_train = dlmread('x_train.csv', ';', 1, 0);
    y_train = dlmread('y_train.csv', ';', 1, 0);

Then proceed to create a regressor object:

.. code-block:: matlab

    builder = py.NeurEco.NeurEcoTabular.Regressor();

The getattr python method can be used to access all the attributes of the model:

.. code-block:: matlab

    version = char(py.getattr(builder, "__version__"));
    path = char(py.getattr(builder, "__path__"));
    methods = char(py.getattr(builder, "__methods__"));
    disp(strcat('version: ', version))
    disp(strcat('path: ', path))
    disp(strcat('methods: ', methods))


.. code-block:: text
    
    version:NeurEco Tabular version 4.01.2474.0 compiled with MSVC v1928  on Oct 12 2022 @ 17:09:04
    path:Dynamic Library loaded from: C:\Program Files\Adagos\NeurEco\bin
    methods:**** NeurEco Tabular Regressor methods: ****
    - load
    - save
    - delete
    - evaluate
    - build
    - get_input_count
    - get_output_count
    - load_model_from_checkpoint
    - get_number_of_networks_from_checkpoint
    - get_weights
    - export_fmu
    - export_c
    - export_onnx
    - export_vba
    - compute_error
    - plot_network
    - forward_derivative
    - gradient
    - set_weights
    - perform_input_sweep

To build the model, first choose the build settings. Note that the build method for a regressor takes only two required arguments, but in this case most of the optional arguments are set:

.. code-block:: matlab

    write_model_to = './EnergyConsumption/EnergyConsumption.ednn';
    checkpoint_address = './EnergyConsumption/EnergyConsumption.checkpoint';
    inputs_shifting = 'min_centered';
    inputs_scaling = 'max_centered';
    outputs_shifting = 'auto';
    outputs_scaling = 'auto';
    inputs_normalize_per_feature = true;
    outputs_normalize_per_feature = false;
    valid_percentage = py.float(33.33);
    use_gpu = false;
    gpu_id = 0;
    checkpoint_to_start_build_from = '';
    final_learning = true;
    disconnect_inputs_if_possible = true;
    initial_beta_reg = py.float(0.1);
    validation_output_data = py.None;
    validation_input_data = py.None;

.. note::
    When passing an argument, the Booleans and the chars are accepted as is. However, when it comes to numerical values the user has to specify the type: py.float, py.int... Same thing when it comes to None (py.None). More information is available about converting between MATLAB types and python types here: `<https://www.mathworks.com/help/matlab/examples.html?category=call-python-libraries&s_tid=CRUX_topnav>`_

The next step is to convert the data from MATLAB doubles to numpy arrays:

.. code-block:: matlab

    n_train_samples = py.int(size(x_train, 1));
    n_inputs = py.int(size(x_train, 2));
    n_outputs = py.int(size(y_train, 2));
    input_train = py.numpy.reshape(py.numpy.array(reshape(x_train.',1,[])), py.tuple({n_train_samples, n_inputs}));
    output_train = py.numpy.reshape(py.numpy.array(reshape(y_train.',1,[])), py.tuple({n_train_samples, n_outputs}));

The data is reshaped twice, because the conversion works only for 1-D arrays, so the 2D double arrays are flattened in MATLAB and reshaped back in python.
Now that the data and the settings are ready, the build method can be called:

.. code-block:: matlab

   builder.build(input_train, output_train, ...
              pyargs('write_model_to', write_model_to, ...
                     'checkpoint_address', checkpoint_address, ...
                     'inputs_shifting', inputs_shifting, ...
                     'outputs_shifting', outputs_shifting, ...
                     'inputs_scaling', inputs_scaling, ...
                     'outputs_scaling', outputs_scaling, ...
                     'outputs_normalize_per_feature', outputs_normalize_per_feature, ...
                     'inputs_normalize_per_feature', inputs_normalize_per_feature, ...
                     'valid_percentage', valid_percentage, ...
                     'checkpoint_to_start_build_from', checkpoint_to_start_build_from, ...
                     'final_learning', final_learning, ...
                     'use_gpu', use_gpu, ...
                     'gpu_id', gpu_id, ...
                     'initial_beta_reg', initial_beta_reg, ...
                     'disconnect_inputs_if_possible', disconnect_inputs_if_possible, ...
                     'validation_output_data', validation_output_data, ...
                     'validation_input_data', validation_input_data)...
              );
   builder.delete();

.. note::
    When passing the optional arguments, we need to use the pyargs method, but no need for that when the arguments are required.

NeurEco will start building the model, and when it's done, the model will be saved in the directory ./EnergyConsumption.
Intermediate models are saved to the checkpoint file, these models are accessible even before the end of the build. The following script show how to load them:

.. code-block:: matlab

   model = py.NeurEco.NeurEcoTabular.Regressor();
    n_models = double(model.get_number_of_networks_from_checkpoint('./EnergyConsumption/EnergyConsumption.checkpoint'));
    for i=1:n_models
        disp(strcat('Loading and evaluating model-', num2str(i), ' from checkpoint file.'))
        model.load_model_from_checkpoint('./EnergyConsumption/EnergyConsumption.checkpoint', py.int(i-1));
        n_trainable_parameters = double(model.get_weights().size);
        disp(strcat('Number of trainable parameters for this temporary model: ', num2str(n_trainable_parameters)))
    end
    model.delete();

.. code-block:: matlab
    
    Loading and evaluating model-1 from checkpoint file.
    Number of trainable parameters for this temporary model:15
    Loading and evaluating model-2 from checkpoint file.
    Number of trainable parameters for this temporary model:29
    Loading and evaluating model-3 from checkpoint file.
    Number of trainable parameters for this temporary model:43
    Loading and evaluating model-4 from checkpoint file.
    Number of trainable parameters for this temporary model:57
    Loading and evaluating model-5 from checkpoint file.
    Number of trainable parameters for this temporary model:57
    Loading and evaluating model-6 from checkpoint file.
    Number of trainable parameters for this temporary model:145
    Loading and evaluating model-7 from checkpoint file.
    Number of trainable parameters for this temporary model:145
    Loading and evaluating model-8 from checkpoint file.
    Number of trainable parameters for this temporary model:145
    Loading and evaluating model-9 from checkpoint file.
    Number of trainable parameters for this temporary model:52

Now let's create a new Regressor object and load the model and extract information about it, such as the number of inputs, the number of outputs and the weights array:

.. code-block:: matlab

    evaluator = py.NeurEco.NeurEcoTabular.Regressor();
    load_status = double(evaluator.load('./EnergyConsumption/EnergyConsumption'));
    if load_status ~= 0
        disp('Loading state = Fail')
    else
        % extracting general information
        disp('Loading state = Success')
        n_inputs = double(evaluator.get_input_count());
        n_outputs = double(evaluator.get_output_count());
        py_weights = evaluator.get_weights();
        weights = double(py.array.array('d', py.numpy.nditer(py_weights)))';
        disp(strcat('Number of Inputs :', num2str(n_inputs)));
        disp(strcat('Number of Outputs :', num2str(n_outputs)));
        disp(strcat('Number of trainable parameters :', num2str(size(weights, 1))));

.. code-block:: text
    
    Loading state = Success
    Number of Inputs :5
    Number of Outputs :1
    Number of trainable parameters :52

.. note::
    When a NeurEco method returns a string, it can be converted by using the char() function, when it returns a numerical it can be converted using the double() function.

For evaluation the following script can be used:

.. code-block:: matlab

    n_test_samples = py.int(size(x_test, 1));
    input_test = py.numpy.reshape(py.numpy.array(reshape(x_test.',1,[])), py.tuple({n_test_samples, py.int(n_inputs)}));
    output_test = py.numpy.reshape(py.numpy.array(reshape(y_test.',1,[])), py.tuple({n_test_samples, py.int(n_outputs)}));
    neureco_outputs_py = evaluator.evaluate(input_test);
    neureco_outputs = reshape(double(py.array.array('d', py.numpy.nditer(neureco_outputs_py))), size(x_test, 1), n_outputs);

Testing data need to be converted to numpy array (like for the build), but the method evaluate will return a numpy array (neureco_outputs_py), so the numpy array need to be transformed into an nditer and then to MATLAB using the method double(). The shape of the output matrix will be lost at this point so it need to be reshaped.
The L2 error can be computed to check how good the model is on the unseen testing data, example:

.. code-block:: matlab

    l2_error = evaluator.compute_error(neureco_outputs_py, output_test);
    disp(strcat('L2 relative error (%) on testing set:', num2str(100 * l2_error)))

.. code:: text

    L2 relative error (%) on testing set:8.567


The model can be saved to a different directory, under a different name:

.. code-block:: matlab

    save_status = double(evaluator.save('EnergyConsumption/NewDir/SameModel'));
    if save_status == 0
       disp('Saving state = Success')
    else
       disp('Saving state = Fail')
    end


.. code:: text

    Saving state = Success


The evaluator can be deleted then a new NeurEco Regressor can be created to export the model (this step is unnecessary, it is proposed for the sake of the code's clarity). The model can be exported to a C-file, an ONNX file, an FMU file or a bas file.

.. code-block:: matlab

    evaluator.delete()
    % Create a model to load and export the NeurEco regressor
    exporter = py.NeurEco.NeurEcoTabular.Regressor();
    load_status = double(exporter.load('./EnergyConsumption/EnergyConsumption'));
    if load_status ~= 0
        disp('Loading state = Fail')
    else
    %     export C Model
        disp('Exporting header File')
        exporter.export_c('./EnergyConsumption/EnergyConsumption.h', 'float');
    %     export ONNX Model
        disp('Exporting ONNX File')
        exporter.export_onnx('./EnergyConsumption/EnergyConsumption.onnx', 'float');
    %     export FMU Model
        disp('Exporting FMU File')
        exporter.export_fmu('./EnergyConsumption/EnergyConsumption.fmu');
    %     export VBA Model
        disp('Exporting VBA File')
        exporter.export_vba('./EnergyConsumption/EnergyConsumption.bas');
    end
    exporter.delete()
